Job systems have hitherto been used in which users operate terminal devices to perform jobs. In such systems, directly describing a program for calling job knowledge or an external program with respect to a graphical user interface (GUI) as an operation object involves reorganization of the job system and has thus been difficult. Should an application programming interface (API) be available for such a job system, the API requests that the user understand each individual API provided for a variety of job systems and set an adapter suitable for an architecture of each job system before the user can perform the above description. The API thus offers poor versatility.
To perform the above description, users have taken an approach of accumulating job knowledge using text or a knowledge sharing tool such as a wiki, or of preparing a script for calling an external program to thereby individually set a shortcut, in addition to the above-described job system.
The users, however, find difficulty in appropriately describing timing at which to refer to the job knowledge in the text described in the knowledge sharing tool in time with a transition from one application window to another. Additionally, because the job knowledge has been converted to primary information such as text, the job system encounters difficulty in providing job knowledge in a format that provides a link with an actual operation on a 2D screen and in referring to an external program with the provision of the job knowledge as an invoker.
In view of the foregoing situation, Non Patent Document 1 discloses a technique with which the job system including no API directly describes a GUI display image within a virtual script to thereby be able to describe an automatic operation program in a simplified manner without modifying an existing system. Non Patent Document 2 discloses a technique that displays text using the foregoing technique. Additionally, Non Patent Document 3 discloses a technique that limits an image recognition target to a forefront display window only and monitors a window handle of the window to thereby solve the problem with Non Patent Document 1. Non Patent Document 4 discloses a technique that sophisticates the annotation offering improved practical usability over the foregoing techniques.